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(S) Video-based system for computer assisted surgery and localisation. 



(57) The apparatus comprises video cameras for 
viewing part of a patient's body and a slide 
projector for projecting a predetermined pat- 
tern of light onto the patient's body which is 
viewed by the cameras. A workstation and video 
frame grabber performs controlling and proces- 
sing functions and are arranged to process said 
images from the cameras using image proces- 
sing algorithms. The images from the cameras 
are superimposed upon prestored images 
which are generated from X-ray or CT scans. A 
passive pointer, having a predetermined pattern 
thereon which is recognisable by the control- 
ling and processing means is used by the sur- 
geon. The pattern is used to identify the exact 
position of a tip of the pointer in relation to the 
patient's body and is displayed on a monitor 
with the superimposed images. 
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This invention concerns assistance to a surgeon 
performing an operation who has planned his work 
from images obtained from pre-operative medical im- 
agers such as X-ray Computer Tomography (CT) or 
Magnetic Resonance (MR) Imagers. The invention's s 
main concern is neurosurgery though other applica- 
tions are possible. At the time of surgery, the surgeon 
would benefit from a means of relating visually detect- 
able patient features to the pre-operatively generated 
images and plans. This process is known as localisa- w 
tion. 

A number of partially successful localisation 
methods have been developed. One set of methods 
is known as frame-based stereotaxy [reviewed by D 
G T Thomas, N D Kitchen: "Minimally Invasive Sur- 15 
gery- Neurosurgery" British Medical Journal Vol 308, 
8 January 1994], In these methods a frame is attach- 
ed rigidly to the patient, by screws into the skull, be- 
fore pre-operative imaging and remains in place in 
surgery. The design of the frame is such that it is visi- 20 
ble in the pre-operative imagery and the surgery is 
planned in terms of measurements relative to the 
frame, so that the plan can be carried out by means 
of scales and graduations marked on the frame and 
visible to the surgeon in-theatre. This approach has 25 
the following disadvantages: 

The need for surgery may not have been deter- 
mined at the time of imaging so a frame may not have 
been in place, in which case second scan would be re- 
quired. 30 

The frame is painful for the patient and there are 
infection risks due to the invasive form of attachment. 
(Some forms of frame have been developed where 
part of the frame is detachable, which makes it slight- 
ly more comfortable but the infection risks are un- 35 
changed. Other forms of frame use bite-blocks (which 
fix via mouldings of the patient's teeth); these are less 
accurate because of the lack of rigidity in the fixing). 

The frame itself can obstruct the surgeon's ac- 
cess, limiting the type of surgery which can be per- 40 
formed. 

The other set of methods are known as frame- 
less or open-stereotaxy. Here, localisation is ach- 
ieved using a patient feature (which is visible in pre- 
operative imagery) or a set of features and a corre- 45 
sponding feature or features locatable intra-opera- 
tively (i.e. in-theatre). These features provide a com- 
mon reference for surgical planning and implementa- 
tion of the plan in-theatre [see Thomas and Kitchen 
above]. These methods generally achieve localisa- so 
tion in the following way: 

The patient's head is clamped using a device 
such as a Mayf ield clamp, to keep it stationary. [The 
Mayf ield clamp is one of a number of devices devel- 
oped for this purpose. This description will assume 55 
use of a Mayf ield clamp, but it is not specific to it]. 

A three-dimensional (3-D) digitiser, sometimes 
referred to as a pointer in this context, i.e. an instru- 



ment which can record the 3D position of its probe tip, 
is used to measure the position of uniquely identifi- 
able patient features relative to some fixed co-ordin- 
ate system in the operating theatre. 

The features of the patient, visible in the pre-op- 
erative imagery, corresponding to those whose pos- 
ition has been measured, as described above, are 
identified by manually marking their position on a 
computer display of the pre-operative data. The com- 
puter generating these displays has access to the out- 
put of the 3D digitiser and the pre-operative images 
and planning data. 

Using the measurements of corresponding 
points, in the coordinate frame of the digitiser and in 
the co-ordinate frame of the pre-operative data, the 
co-ordinate transformation (between these two 
frames of co-ordinates) is calculated by a computer 
program. 

With the transformation between digitiser co-or- 
dinates and the pre-operative image co-ordinates 
known, the transformed position of the digitiser 
probe-tip, or a surgical instrument attached to the 
digitiser, can be plotted on computer displays of the 
pre-operative data or display, enabling the surgeon to 
locate his instruments to some pre-planned position 
or identify on the pre-operative data some visually 
observed feature, revealed during surgery. 

The existing frameless stereotactic methods 
have disadvantages arising from restrictions due to 
nature of the 3D digitiser employed. A number of dif- 
ferent techniques have been developed. [For refer- 
ences see A C F Colchester et al, "VISLAN: Combin- 
ing Intra-Operative Video and Pre-Operative Images 
for Surgical Guidance" to be published in Proc. Appli- 
cations of Computer Vision in Medical Image Proc- 
essing Ed: G M Wells III, Menlo Park, AAAI Spring 
Symposium March 1994, AAAI Press]. Asystem, with 
a mechanical arm digitiser, is inherently difficult to 
manoeuvre and is time consuming to use if numerous 
features must be located to achieve registration; sys- 
tems using acoustic or LED beacons require the 
hand-held pointer of the digitiser to have trailing wires 
or an internal power supply resulting in complexity in 
design and certification. Other disadvantages of the 
existing approaches are: 

Loss of accuracy resulting from errors in the man- 
ual identification of patient features. (These can be 
reduced by the use of markers attached to the pa- 
tient, if this is done before pre-operative imaging, but 
this carries risks of infection with markers held by 
screws into bone or errors due to skin movement if 
markers are attached to the skin by glue). 

Loss of localisation capabilities, or errors, due to 
accidental or deliberate movement of the patient. The 
problem can be reduced by including a means of 
measuring the position of the clamp holding the pa- 
tient, but this requires the patient to remain fixed in 
the clamp throughout surgery. 
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An aim of the present invention is to provide ap- 
paratus and a method of achieving localisation in 
f rameless stereotaxy which does not suffer from the 
above mentioned problems. 

According to the present invention there is provid- s 
ed apparatus for computer assisted surgery compris- 
ing camera means for viewing part of a patient's body, 
projector means for projecting a predetermined pat- 
tern of light onto the patient's body which is viewed by 
said camera means, processing and controlling 10 
means including image processing algorithms ar- 
ranged to process said images from said camera 
means, and superimpose said images on images 
generated from data prestored in said controlling and 
processing means, display means for displaying said 15 
images and superimposed images, a passive pointer 
means having a predetermined pattern, which is 
viewed by said camera means and recognisable by 
said controlling and processing means, which is ar- 
ranged to determine the position of a tip of said poin- 20 
ter means and orientation of said pointer means in re- 
lation to said patient's body and display said position 
on said display means with respect to said images 
generated from said data prestored in said controlling 
and processing means. 25 

According to the present invention there is provid- 
ed a method for use in computer aided surgery com- 
prises the steps of: 

generating at least one three dimensional im- 
age from a CT, MR or X-ray scan of part of a patient's 30 
body, projecting a predetermined pattern of light onto 
said part of said patient's body, viewing said pattern 
by camera means, generating a three dimensional 
image from said viewed pattern of light combining 
said three dimensional images to give a common 35 
frame of reference, and, using a passive pointer 
means having a predetermined pattern enabling 
processing algorithms to determine the position of a 
tip of said passive pointer means and orientation of 
said pointer means in relation to the patient's body, 40 
and thereby identify the position of said pointer 
means in relation to the image generated from the CT, 
MR or X-ray scan. 

An embodiment of the present invention will now 
be described with reference to the accompanying 45 
drawings, wherein, 

FIGURE 1 shows a pictorial diagram of the appa- 
ratus required to perform the invention; 
FIGURE 2 shows the patterned slide used in the 
projector shown in Figure 1 ; so 
FIGURE 3 shows a pointer handle and probe, 
with an alternative linear acquisition target, and, 
FIGURE 4 shows an intra-operative marker pin 
connected to bone. 

Referring to the drawings, the invention as de- 55 
scribed provides an alternative method of achieving 
localisation in frameless stereotaxy by using video 
cameras and a structured light projector to obtain in- 



tra-operative information about the position of patient 
features and of surgical instruments or pointers. The 
advantages of this method are that: 

The video-based pointer can be freely hand-held 
without trailing wires or mechanical encumbrances. 

The pointer does not contain active components, 
simplifying its design, certification, and sterilisation. 

The position and shape of the surface feature, 
e.g. the patient's skin, can be measured very quickly 
using the cameras and projector without the laborious 
use of the 3D pointer. 

The position of other features of the patient, such 
as blood vessels revealed during surgery, can be 
measured quickly with cameras alone, to provide a 
means of refining the transformation between the vid- 
eo camera system and the pre-operative images and 
plans. 

The system does not require the use of special at- 
tachments to the patient during pre-operative imag- 
ing. 

The errors or loss of capability due to accidental 
ordeliberate movement of the patient can be detected 
and corrected by a number of methods, e.g. the use 
of markers attached intra-operatively to the patient, 
or supporting frame or operating table, at positions of 
convenience to the surgeon, or the use of patient pos- 
itioning based on patient features such as revealed 
blood vessels. 

As surgery progresses, the accuracy of registra- 
tion to be refined is permitted using revealed patient 
features such as blood vessels. 

Considering now the system components, and 
referring to Figure 1, the system consists of the fol- 
lowing main components. 

A workstation computer 2 with processor is pro- 
vided having a display monitor and keyboard and/or 
mouse for normal computer interaction. An internal 
loudspeaker is also required for some system func- 
tions. Pre-operative data, images and surgical plans 
are loaded into this machine before surgery starts. A 
video frame grabber/video display card 4 is hosted by 
the workstation 2. At least one or more video cameras 
6 are connected to the frame grabber. (In a different 
configuration, the cameras might be attached to, or 
use the optics of, the surgeon's operating micro- 
scope). A structured light projector 8 and slide is also 
provided. The projector is essentially a conventional 
slide projector, with appropriate lens to ensure the re- 
quired field of projection and focus. The projected 
slide is as shown in Figure 2. The breaks in the vert- 
ical bars are required to simplify processing algo- 
rithms. Different densities of bars may be used and 
different slides or more (or less) closely spaced bars 
and with different patterns or breaks may be used de- 
pending on the spatial resolution required. Astand 10 
is provided for the cameras and projector. The stand 
is required to position the cameras and projector in a 
suitable position for viewing the relevant part of the 
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patient. The stand consists of two main parts. Firstly, 
a base 1 6 and arms 1 8. The stand and arms are used 
to position the cameras 6 and projector 8 in the cor- 
rect general viewing position. This is usually above 
the patient's chest or head for an operation on the 5 
head. In plan view, the surgeon, the surgical wound 
and the cameras should be placed in a straight line 
approximately. Secondly, a camera and projector 
mount 12. The mount 12 allows the position of the 
field of view of the cameras 6 and projector 8 to be ad- 10 
justed from the general viewing position set by the 
placement of the stand and arms, using a pan-and-tilt 
device. The mount should allow a range of alternative 
spacings of cameras and projector and alternative 
vergeance angles so that the set-up can be adjusted is 
for the particular operation in question. A video dis- 
play monitor 1 4 is connected to the frame grabber and 
display card 4. Acamera calibration tile 20 is provided 
for calibration purposes. The camera calibration tile is 
a flat plate with an accuracy measured array of dots 20 
on it. The calibration tile and its use are described by 
C G Harris and A Teeder "Geometric Camera Calibra- 
tion for Vision-Based Navigation", Proceedings of 
IFAC Internationa] Workshop on Autonomous Vehi- 
cles, 18-21 April 1993, Southampton UK, pp 77-82. A 25 
video pointer handle and probe 22 is a hand-held in- 
strument The hand-held component of the video-ba- 
sed pointer consists of a patterned handle with a 
probe-tool attachment 32 such that the probe is rig- 
idly fixed with respect to the handle in a known posi- 30 
tion when in use. Use of a bayonet fitting or similar 
fitting between handle and probe enables different 
tools to be attached to the handle (knives, suckers) for 
different purposes. The patterned handle is marked 
as shown in Figure 3. The handle has finger-grips 35 
positioned, either at the side of it or behind the pat- 
terned face-plate, so that the surgeon's fingers do not 
obscure the patterning when the pointer is in use, 
though the system can work with partial obscuration 
of the patterning. The patterning consists of two com- 40 
ponents. Firstly, a Binary Acquisition Target (BAT) 26. 
This is a symbol designed for quick detection and ori- 
entation by image processing algorithms. The shown 
symbol is one possibility, but there are others, e.g. the 
linear BAT 28 also shown in Figure 3. This lineartarget 45 
may also be used for tracking purposes. Secondly, a 
Tracking Pattern 30. This is a set of marking designed 
to be suitable for video tracking purposes using the 
tracking algorithms, as described in GB Patent Appli- 
cation Number 9114518.5. The drawn pattern is one so 
example as many variations of this are possible. 

The Intra-Operative Markers (lOMs) 24, Figure 1 , 
may take several forms, though in all cases the mark- 
ers need to be rigidly attached (directly or indirectly) 
to the patient when in use. Individual markers may be 55 
attached separately to the patient by screws which 
screw into the bone. These markers consists of three 
parts, a base section 34 which screws into the bone 



and a post 36 and a head 38, which screws into the 
base, as shown in Figure 4. (The separable design al- 
lows the post and head to be removed temporarily 
and later replaced in the same position if it obstructs 
some part of the surgical procedure. Alternatively, a 
one-piece marker might be used where the surgical 
procedure allows). A minimum of three such markers 
are required. The head of the marker is finished in a 
bright matte finish to provide good visual contrast 
against its background. The shape of the marker-he- 
ad is to enable it to be quickly and accurately located 
by image processing algorithms, thus a spherical 
head is very suitable, but other designs (e.g. ellip- 
soids, circular discs) might be used instead. For ex- 
ample, individual markers may be used which attach 
separately at convenient points to the Mayfield 
clamp, or similar clamp holding the patient, or to a 
separate dedicated clamp attached to the patient, or 
to the operating table. The heads of these markers 
are as described above. These markers may also be 
made in two parts for temporary removal, and may in- 
clude a patterned block, containing a BAT and a 
tracking pattern similar to that used on the pointer 
handle. This is attached directly to the patient by a 
mechanism similar to a Mayfield clamp. As described 
above, part of the marker may be detachable for ease 
of use, and may include a patterned block as descri- 
bed above, but attached to the Mayfield clamp hold- 
ing the patient, or to the operating table. 

Considering now the system operation, the sys- 
tem is operated in the following stages which are de- 
scribed in the context of a surgical craniotomy. 

The first step is camera set-up and calibration. 
With the patient lying on the operating table and 
clamped, if part of the surgical procedure, but not yet 
draped for surgery, the cameras 6 and projector 8 are 
positioned and adjusted to provide a suitable field of 
view. The calibration tile 20 is then held in the over- 
lapping fields of view of the cameras and a set of pairs 
of images are captured, each pair of images consists 
of a simultaneously captured image from each cam- 
era. 

The next step is the use of calibration algorithms, 
as described by Harris and Teeder above, running on 
the workstation 2, to calculate the relative position 
and orientation of the cameras 6 and parameters de- 
scribing the imaging geometry of each camera (e.g. 
focal length, aspect ratio, image distortions). This 
process must be repeated if the cameras are moved 
relative to each other or if their imaging geometry is 
disturbed, but otherwise the cameras can be moved 
to a different view point or to view a different direction 
without repeating the calibration procedure. 

The next step is the intra-operative skin surface 
measurement. Using the projector 8, the bar-pat- 
terned slide (Figure 2) is projected onto the patient's 
skin and a pair of images is captured from the cam- 
eras. Using a processing algorithm running on the 
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workstation 2, the shape and position of a section of 
skin surface illuminated by the projector is calculated 
relative to a particular camera. The processing will 
now be described. 

A dense and accurate 3D representation of the 5 
(shaven) face or upper head is needed to permit 
matching to the surface extracted from MR or CT im- 
agers. Accuracy's of better than 1mm are believed re- 
quired. Note that the upper head will consist of a sin- 
gle smooth, continuous surface. 10 

Measurements of the surface will be performed 
using one or more cameras, for speed of acquisition 
and density of results. The surface is by itself devoid 
of prominent visual features, so visual features must 
be imposed on the surface. To obtain high accuracy is 
(sub-pixel), support from a local region of the image 
is needed. One option for the surface features is a 
fine texture, but this has the disadvantages of vari- 
able accuracy, and the need to know the local surface 
slope (which will stretch the observed texture). 20 

The preferred option is to use dark-light edges, 
imposed on the surface either by projection or 'pain- 
ting'. These can be consistently located to high accu- 
racy, and can be relatively dense. A Canny-type proc- 
ess is used, with a smoothing mask of size smooth 25 
(bold type is used to note algorithm parameters), and 
up to Nhysteresis steps of hysteresis to grow from 
the high threshold edgels into the low threshold 
edgels. The thresholds are obtained adaptively by 
taking the first moment of the distribution of edge 30 
strengths, the low threshold being a percentage of the 
high threshold. The density is limited by the size of the 
smoothing mask used in edge extraction - placing the 
edges closer together then this wilt cause the edges 
to be mis-positioned, as the support mask samples 35 
neighbouring edge structures. Note that edges can- 
not be accurately located at junctions and sharp 
bends, and for this reason, a number, safety, of the 
first and last edgels of an edge are discarded. Edges 
shorter than mln Jength are discarded. Perhaps the 40 
strongest disadvantage of dark-light edges is the pos- 
sibility of incorrect labelling of the edges - they have 
no distinguishing attributes excepting their polarity. 
This could be overcome by use of a chequerboard, but 
measurement accuracy will be lost near the junctions. 45 

Three-dimension measurements of the location 
of surface features will be performed using stereo. 
One option is to use calibrated structured light, ob- 
served by a single camera, but this presents the prob- 
lem of how to calibrate the structured light, and how 50 
stable is the calibration over time. The alternative se- 
lected is to use uncalibrated (but structured) surface 
edge features, observed by binocular (or trinocular) 
stereo cameras. Stereo calibration is essentially a 
solved problem, though there could be some diff icul- 55 
ties with telephoto lenses. Using a conventional bi- 
nocular stereo configuration, with the cameras 
spaced apart horizontally, the dark-light edges that 



give the greatest accuracy will be perpendicular to the 
epi-polar lines, thus they should be largely vertical in 
the images. Edges that are parallel with the epi-polar 
lines give no depth information, and this is another 
point against using a chequerboard. 

Using stereo on dark-light edges requires the cor- 
rect labelling of edges in each image, as there is no 
local test to check that the correct edges have been 
matched. The labelling problem is solved by firstly ob- 
taining the correct relative labelling on each image in- 
dividually, and then finding the difference in labelling 
between the two images. 

To obtain the labelling on a single image, the ad- 
jacency between edges is first determined by send- 
ing out a set of 'feelers' on the image perpendicular to 
the edge, and noting which edges they first collide 
with. The feelers are sent out at an interval of poke- 
step between edgels along an edge, and have max- 
imum length poke-length. The pair of edges with the 
greatest commonality are first labelled as 1 and 2. 
Next, the edge with the greatest commonality with the 
previously labelled edges is found, and appropriately 
labelled. This continues until no new edges are lab- 
elled. 

The difference in labelling between the two im- 
ages is currently determined by introducing randomly 
positioned breaks in the edges, which are seen as the 
start and end of edges. Each break should be able to 
be epi-polar matched between the images, though 
great accuracy is not expected, as it may be on a 
sharp curve. The epi-polar matching is deemed to be 
successful if it is within a distance of epi-polar- 
thresh pixels. This distinguishes between valid and 
invalid labelling differences. The labelling difference 
which is most popular is selected. 

Once labelling has been achieved, each labelled 
pair of edges are considered separately. For each 
edgel in the prime camera (camera 0), the epi-polar 
line is constructed in the secondary camera image, 
and the pair of consecutive edgels in the secondary 
image that straddle the epi-polar line are found. By in- 
tersecting the epi-polar line and the line joining the 
straddling edgels, the intersection point is found. Now 
reciprocal depth varies linearly along the epi-polar 
line (in homogeneous co-ordinates). The start of the 
epi-polar line is location of the prime camera pin-hole, 
and the end of the epi-polar line has a reciprocal 
depth of zero. As before, a number, safety, of the first 
and last edgels of an edge are discarded because 
they may be inaccurate. To provide further security, 
the local orientation of both edges must be within an 
angle of epl-polar-angle degrees to the vertical. 

The 3D edgels can be used to construct a trian- 
gular planar facet surface by use of Delaunay trian- 
gulation. Note every edgel is needed, since they are 
correlated, and edgels at an interval of delaunay- 
step are selected. Delaunay triangulation is per- 
formed in the prime image, and the resulting triangles 
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interpreted as planar facets. This produces a single- 
valued surface. Any triangle with an edge exceeding 
max-triangle-slze in length is discarded to stop the 
interpolation becoming too distant from the observa- 
tions. 5 

The next step is the registration of intra-opera- 
tively and pre-operatively measured skin surfaces. 
Using a processing algorithm, running on the work- 
station 2, the shape of the skin measured, as descri- 
bed above, is registered to the shape of the corre- 10 
sponding section of skin previously generated by ana- 
lysis of pre-operative data. This results in the calcu- 
lation of the co-ordinate transformation, T, between 
the pre-operative data and positions measured in the- 
atre relative to the specified camera, as described 15 
above. 

The processing algorithm to match the pre-oper- 
ative and intra-operatively measured surfaces may 
be based on chamfer match procedures (D G L Hill 
and D J Hawkes, "Medical image registration using 20 
knowledge of adjacency of anatomical structures", 
Image and Vision Computing 12(3) 1994, in press). In 
this technique, a distance transform is calculated for 
the surface extracted from the pre-operative data, 
and is held as a set of voxels whose value represents 25 
the shortest distance to the pre-operative measure- 
ment of the scalp surface. A series of trial registra- 
tions are then attempted, by projecting points on the 
intra-operatively measured skin into the pre-opera- 
tive co-ordinates. For each trial registration, the dis- 30 
tance values of the voxels addressed by the projected 
points are summed to produce a cost of the trial. The 
final registration is taken as the minimum cost pose, 
found by using a genetic algorithm to refine an initial 
coarse estimate. Alternative methods are possible. 35 

The next step is the marking of planned craniot- 
omy position. With the patient and cameras remaining 
in the positions used, as described above, the sur- 
geon is assisted in marking on the patient the planned 
site of craniotomy. This may be done in several ways. 40 
The live video output of a selected camera 6 is dis- 
played to the surgeon with a graphical overlay gener- 
ated by the frame grabber/video display 4. The over- 
lay shows the outline of the planned craniotomy as if 
seen from the position of the selected camera. The 45 
position of this overlay can be calculated in the work- 
station, from its planned position relative to the pre- 
operative data and the transformation, T. The sur- 
geon now marks the skin surface manually, in the nor- 
mal way, but he looks at the video display to ensure 50 
that the marker pen and plan are in alignment The 
marker pen used by the surgeon is fitted to the pat- 
terned handle of the video pointer 22. Using the video 
pointer 22, the position of the marker-tip is now 
tracked relative to the camera 6, and, by means of the 55 
transformation, T, its position relative to pre-opera- 
tive images and plans can be calculated. The surgeon 
may now mark the planned craniotomy on the patient, 



by observing the position of the pen-tip on a display 
of pre-operative data or by listening to a audio signal 
indicating the proximity of the pen-tip to the planned 
position. 

The next step is the placement and modelling of 
intra-operative markers 24. Having marked the cra- 
niotomy position on the patient the surgeon now 
places lOMs 24 at positions of convenience (but with- 
in the field of view of the cameras 6), either attaching 
a patterned block or individual markers to the May- 
field clamp, or operating table or directly to the pa- 
tient, depending on the type of IOM in use. 

The position of the markers 24 and position and 
shape of the skin surface is now measured, relative 
to the camera system. The skin surface is measured, 
as described above. (This step can be omitted if the 
position of the cameras and patient has remained un- 
changed since that step was performed). A pair of im- 
ages is also captured, without the patterned illumin- 
ation from the projector 8. These images are process- 
ed in the workstation to determine the layout and pos- 
ition of marker pin heads (if separate markers were 
used) or to determine the position of the IOM block (if 
a block were used). With independent markers, the 
position of each marker head is calculated relative to 
the camera system. The processing algorithms will 
now be described. 

The pin tracking module is designed to find 
movement of the patient relative to the camera 6. The 
program works in two modes, 'create* and find'. In 
create mode, the individual marker pin positions are 
measured and stored in a file as the model with which 
pin positions are compared after patient-camera 
movement After movement, find mode is operated, in 
which the new 3D pin positions are measured, and the 
old and new positions compared to calculate the rel- 
ative movement. 

Create Mode 

The steps in create mode are: 

Two images are captured and stored in arrays. 

Circles are found in each image as follows. A local 
mean removal (tMR) algorithm is applied which sub- 
tracts off the local background intensity, so removing 
noise and enhancing edges. The tMR code then as- 
signs a class number to each pixel, by comparing it 
with pre-set thresholds. This 'class* image is then 
stored as the working image. For each working im- 
age, a connected component finder algorithm is ap- 
plied, which searches for groups of connected pixels 
of the same dass, and labels each group as an object 
It also finds the outline pixels for each object. Each 
object is examined to see if it satisfies the criteria for 
being a candidate to be a circle, i.e. width, height, and 
class are checked. Candidate circles are checked by 
the circle-finding routine, which fits a circle to the ob- 
ject outline, using a general least squares minimisa- 
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tion routine, and returns its radius, centre and error 
in fit Those with acceptable errors are stored as a list 
of circles. Matching circles in the two images are iden- 
tified using epi-polar matching, and their 3D positions 
calculated, using the known camera position. The 3D 
positions are stored as the pin model. 

Find Mode 

Steps 1-3 in 'find mode' are identical to those descri- 
bed above. The 3D positions of the pins are stored in 
an array to form the object to which the pin model is 
to be matched. The pin model is read in and the two 
sets of pin positions passed to a closed-form least- 
squares minimisation algorithm which finds the best- 
fit motion of the model about the camera position to 
bring it onto the object position. It is necessary to tell 
the algorithm which model pin matches which object 
pin before the fit is performed. In practice, as the pins 
are placed in a circle, it is easy to order them accord- 
ing to angle about their centroid giving N possible 
match arrangements for N pins. 

With an IOM block, whose shape and patterning 
is already known, the tracking algorithms may be 
used to determine its position as hereinater descri- 
bed. 

The intra-operative and pre-operative measure- 
ment of the skin surface are now registered again, as 
described above, and the resulting updated transfor- 
mation, T, is used to calculate the position of the 
markers relative to the pre-operative images and 
plans. This information is stored for future use follow- 
ing relative movement of camera and patient. 

At this point surgery now commences, and at va- 
rious times the surgeon may wish to use the video-ba- 
sed pointer 22 (or its handle with another instrument 
attached) to help relate some intra-operative position 
to the pre-operative data. This may be done by real- 
time processing of images in two phases as follows: 

Acquisition - The surgeon holds the pointer 22 
still, with its patterning facing the cameras. Process- 
ing algorithms locate the BAT in corresponding im- 
ages from each camera and an approximate estimate 
of the position of the BAT, relative to the cameras is 
calculated. This position is used as an initial estimate 
for the tracking phase. Successful completion of this 
phase is indicated by an audible signal and the sur- 
geon may start to move the pointer. The processing 
algorithm for BAT acquisition will now be described. 

Image processing algorithms for locating the bi- 
nary acquisition target, shown in Figure 3, proceed in 
a series of steps. 

The first steps segment potential BAT objects 
from the field of view, as follows: 

1) The input grey-level image is processed by lo- 
cal mean removal and thresholding to produce a 
binary output image. 

2) Connected (black-pixel) components are then 



found. The following steps identify the BAT from 
the set of connected components. 

3) The set of components arising from step 2 are 
filtered according to component size to remove 

5 very small items resulting from image noise and 

clutter. 

4) The convex hull of each remaining connected 
component is calculated. 

5) Significant concavities in object boundaries 
10 are found by comparing the true boundary with 

the convex hull, and objects with other than three 
major concavities are rejected. 

6) Again by comparing the true boundary with the 
object's convex hull, the six points marking the 

15 end of each concavity are found. 

7) The distances between opposite pairs of pints 
from step 6 are compared. In orthographic pro- 
jection, these distances would be equal for the 
true BAT, because the expected convex hull is 

20 hexagonal, consequently objects with signifi- 
cantly unequal opposite distances can be reject- 
ed. In practice the comparison of distances al- 
lows for the fact that the viewing geometry has 
perspective and is not simply orthographic. The 

25 remaining steps calculate the pose of the BAT. 

8) The position and orientation of the BAT, rela- 
tive to the camera, is now calculated from the im- 
age positions of the six points defining the near- 
hexagonal convex hull, ignoring the rotational 

30 ambiguity, using knowledge of the actual dimen- 

sions of the BAT and the camera calibration para- 
meters. 

9) With three possibilities for the pose of the BAT 
calculated in step 8, differing only by 120 degree 

35 rotations of the BAT, the rotational ambiguity is 
now resolved by calculating the position of three 
test pixels, to locate the deepest cut in the BATs 
hexagonal convex hull. 

This process can be repeated for images from 

40 each of the cameras and the resulting set of pose es- 
timates can be integrated for improved accuracy. 

Tracking - Using the BAT position estimate calcu- 
lated at acquisition, the tracking pattern is tracked us- 
ing the tracking algorithms as described in GB Patent 

45 Application Number 9114518.5. The accuracy of this 
phase is improved by an extension of the tracking al- 
gorithms to use information from two or more cam- 
eras 6. Using the transformation, T, the position of the 
pointer 22 or tool-tip can be superimposed on dis- 

50 plays of pre-operative images or plans. As the track- 
ing algorithms track its target in orientation as well as 
position, the orientation of the pointer can be dis- 
played. Should the system fail to track the pointer, this 
can be detected by the system and an audible warn- 

55 ing generated. 

Registration of intra-operativety and pre-opera- 
tively measured patient features revealed in surgery. 
In the course of surgery, other patient features will be 
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revealed to the intra-operative cameras. A particular 
example is the network of pial blood vessels revealed 
after removal of the dural membrane. Depending on 
the intra-cranial pressure, tissue may rise or fall on 
opening the craniotomy, and tissue may be misplaced 5 
relative to its pre-operative position. The intra-oper- 
ative position and shape of these blood vessels can 
be measured by processing a new pair of images and 
the feature registered to pre-operative measure- 
ments. This registration provides a second estimate 10 
that transformation, T. This second estimate may be 
used to replace or refine the original estimate of T. 
Similar techniques may be applied to other patient 
features. 

Considering nowthe re-location of the patientfol- 15 
lowing movement of patient or cameras. From time to 
time, it may be necessary to move the cameras or the 
patient, fbrthe convenience of the surgeon to improve 
surgical access or line of sight, or movement may oc- 
cur for some accidental reason. Following such move- 20 
ment, the transformation, T, must be re-calculated. If 
the skin surface is no longer visible, for example, be- 
cause of surgical wadding and drapes, this may be 
done by capturing a new pair of images containing the 
lOMs 24 (without projector illumination). Where the 25 
lOMs 24 consists of a patterned block, its new intra- 
operative position may be measured, as described 
above, and T can be updated because the lOM's pos- 
ition relative to pre-operative data has been calculat- 
ed, as described above. Where individual markers are 30 
used, the newly captured pair of images is used to 
make a second measurement of the marker head pos- 
itions and a further registration algorithm estimates 
the best common transformation, T, between original 
and current IOM positions. The processing algorithms 35 
for this operation have been described above with re- 
spect to the placement of the lOM's. The transforma- 
tion, T, between the camera system and pre-opera- 
tive images and plans can now be revised. 

The sources of movement which can be accom- 40 
modated, depend on the type and fixing of IOM used. 
All the types described above can be used to accom- 
modate deliberate movement of the operating table or 
camera and projector mount. If movement occurs ac- 
cidentally because the head slips in the Mayf ield sup- 45 
port clamp, then recovery from this situation requires 
the lOMs to have been fixed directly to patient 

Intra-operative visualisation of pre-operative 
plans may be required by the surgeon at various sta- 
ges in surgery. This may be done, as previously de- so 
scribed, by overlaying transformed planning data on 
live video from a selected camera. 

It will be appreciated that various modifications 
may be made to the foregoing described procedure 
and apparatus, are possible which fall within the 55 
scope of the invention. For example, any number of 
cameras may be used, and variations to the algo- 
rithms used may be envisaged. 



Claims 

1. Apparatus for computer assisted surgery com- 
prising camera means for viewing part of a pa- 
tient's body, projector means for projecting a pre- 
determined pattern of light onto the patient's 
body which is viewed by said camera means, 
processing and controlling means including im- 
age processing algorithms arranged to process 
said images from said camera means, and super- 
impose said images on images generated from 
data prestored in said controlling and processing 
means, display means for displaying said images 
and superimposed images, a passive pointer 
means having a predetermined pattern, which is 
viewed by said camera means and recognisable 
by said controlling and processing means, which 
is arranged to determine the position of a tip of 
said pointer means and orientation of said pointer 
means in relation to said patient's body and dis- 
play said position on said display means with re- 
spect to said images generated from said data 
prestored in said controlling and processing 
means. 

2. Apparatus as claimed in Claim 1, wherein said 
camera means comprises two or more video 
cameras to provide a three-dimensional image. 

3. Apparatus as claimed in Claim 2, wherein calibra- 
tion means is provided for calibrating said video 
cameras. 

4. Apparatus as claimed in any preceding claim, 
wherein the pointer means comprises a handle 
having a binary acquisition target thereon for rec- 
ognition and orientation by image processing al- 
gorithms. 

5. Apparatus as claimed in claim 4, wherein said 
handle includes a tracking pattern used for video 
tracking by tracking algorithms. 

6. Apparatus as claimed in Claim 5, wherein the 
pointer means includes means for attaching dif- 
ferent tools, the position of a tip of each tool and 
orientation of the tool, in relation to the patients 
body being determined by said controlling and 
processing means. 

7. Apparatus as claimed in any preceding claim, 
wherein said projector means is a slide projector 
and said slide comprises a pattern of broken vert- 
ical bars, which may be of different densities. 

8. Apparatus as claimed in any preceding claim, 
wherein the images generated from prestored 
data are formed from X-ray images, MR or CT 
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scan images. 

9. Apparatus as claimed in any preceding claim, 
wherein marker pins are attached to the part of 

the patients body to be viewed, and used by said s 
controlling and processing means to compensate 
for movement of the patient's body. 

10. Apparatus as claimed in Claim 9 f wherein the 
marker pins comprise a base section adapted to 10 
be screwed into the patients bone, a post and a 
head, said head being adapted to be easily rec- 
ognised by said image processing algorithms. 

11. Apparatus as claimed in Claim 10, wherein said 15 
head includes a pattern for recognition by said 
image processing algorithms. 

12. Apparatus substantially as hereinbefore descri- 
bed with reference to the accompanying draw- 20 
ings. 

13. A method, for use in computer aided surgery 
comprising the steps of: 

generating at least one three dimensional 25 
image from a CT, MR or X-ray scan of part of a 
patient's body, projecting a predetermined pat- 
tern of light onto said part of said patient's body, 
viewing said pattern by camera means, generat- 
ing a three dimensional image from said viewed 30 
pattern of light, combining said three dimensional 
images to give a common frame of reference, 
and, using a passive pointer means having a pre- 
determined pattern enabling processing algo- 
rithms to determine the position of a tip of said 35 
passive pointer means and orientation of said 
pointer means in relation to the patient's body, 
and thereby identify the position of said pointer 
means in relation to the image generated from 
the CT, MR or X-ray scan. 40 

14. A method as claimed in Claim 13, wherein said 
camera means comprises at least two cameras 
which are calibrated so that each camera cap- 
tures the same image, and their relative position 45 
and orientation are calculated. 

15. A method as claimed in Claim 13 or Claim 14, 
wherein marker pins are positioned on the pa- 
tient's body, so that any movement thereof is de- so 
tected and compensated for. 



9 



EP0 672 389 A2 




10 



EP0 672 



389 A2 




Fig. 2 
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Fig. 3 
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